The increasing size of language models raises great research interests in parameter-efficient fine-tuning such as LoRA that freezes the pre-trained model, and injects small-scale trainable parameters for multiple downstream tasks (e.g., summarization, question answering and translation). To further enhance the efficiency of fine-tuning, we propose a framework that integrates LoRA and structured layer pruning. The integrated framework is validated on two created deidentified medical report summarization datasets based on MIMIC-IV-Note and two public medical dialogue datasets. By tuning 0.6% parameters of the original model and pruning over 30% Transformer-layers, our framework can reduce 50% of GPU memory usage and speed up 100% of the training phase, while preserving over 92% generation qualities on free-text sequence-to-sequence tasks.

Barcelona, Spain – 28 Aug 2022: A novel artificial intelligence (AI) algorithm uses routine echocardiograms to identify aortic stenosis patients at high risk of death who could benefit from treatment. The late breaking research is presented in a Hot Line session today at ESC Congress 2022.1 Aortic stenosis is the most common primary valve lesion requiring surgery or transcatheter intervention in Europe and North America.2 Prevalence is rapidly increasing due to ageing populations. Guidelines strongly advise early intervention in all symptomatic patients with severe aortic stenosis due to the dismal prognosis. Approximately 50% of untreated patients with aortic stenosis die in the first two years after symptoms appear.3